Grading Corneal Fluorescein Staining

ABSTRACT

The technology described in this document can be embodied in systems and computer-implemented methods for determining a score representing an amount of staining of the cornea. The methods include obtaining a digital image of the cornea stained with a tracer material, receiving a selection of a portion of the image, and processing, by a processing device, the selection to exclude areas with one or more artifacts to define an evaluation area. For each of a plurality of pixels within the evaluation area, a plurality of Cartesian color components are determined and a hue value in a polar coordinate based color space is calculated from the components. An amount of staining of the cornea is then determined as a function of the hue value. The methods also include assigning a score to the evaluation area based on the amount of staining calculated for the plurality of pixels.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/016,898, filed Jun. 25, 2018, and issuing on Dec. 3, 2019 as U.S.Pat. No. 10,492,674, which is a continuation of U.S. patent applicationSer. No. 15/308,184, filed Nov. 1, 2016, now U.S. Pat. No. 10,004,395,which is a 371 U.S. National application of PCT/US2015/028907, filed May1, 2015, which claims priority to U.S. Provisional Application61/988,144, filed on May 2, 2014. The entire contents of the foregoingare incorporated herein by reference.

FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under Grant No. K24EY019098 awarded by the National Institutes of Health. The Governmenthas certain rights in the invention.

TECHNICAL FIELD

This disclosure relates to quantitative determination of a particularcolor content in an image or a portion of the image.

BACKGROUND

Clinicians often grade the amount of fluorescence in corneal images ofpatients as a measure of corneal epithelial disease.

SUMMARY

In one aspect, this document features computer-implemented methods fordetermining a score representing an amount of staining of the cornea.The methods include obtaining a digital image of the cornea stained witha tracer material, receiving a selection of a portion of the image, andprocessing, by a processing device, the selection to exclude areas withone or more artifacts to define an evaluation area. For each of aplurality of pixels within the evaluation area, a plurality of Cartesiancolor components are determined and a hue value in a polarcoordinate-based color space is calculated from the components. Anamount of staining of the cornea is then determined as a function of thehue value. The methods also include assigning a score to the evaluationarea based on the amount of staining calculated for the plurality ofpixels.

In another aspect, this document features systems for determining ascore representing an amount of staining of the cornea. The systemsinclude an imaging system configured to obtain a digital image of thecornea stained with a tracer material, and a score calculator module.The score calculator module is configured to receive a selection of aportion of the image via a user interface, and process the selection toexclude areas with one or more artifacts to define an evaluation area.For each of a plurality of pixels within the evaluation area, the scorecalculator module determines a plurality of Cartesian color componentsand calculates a hue value in a polar coordinate-based color space basedon the components. The score calculator module also determines an amountof staining of the cornea as a function of the hue value, and assigns ascore to the evaluation area based on the amount of staining calculatedfor the plurality of pixels.

In another aspect, this document features computer readable storagedevices that have encoded thereon computer readable instructions. Theinstructions, when executed by a processor, cause a processor to performvarious operations. The operations include obtaining a digital image ofthe cornea stained with a tracer material, receiving a selection of aportion of the image, and processing the selection to exclude areas withone or more artifacts to define an evaluation area. For each of aplurality of pixels within the evaluation area, a plurality of Cartesiancolor components are determined and a hue value in a polarcoordinate-based color space is calculated from the components. Anamount of staining of the cornea is then determined as a function of thehue value. The operations also include assigning a score to theevaluation area based on the amount of staining calculated for theplurality of pixels.

In another aspect, this document features methods of determiningseverity of an eye condition. The methods include determining a scorefor a cornea of a subject using the method described above. Thedetermined score can indicate the severity of the condition.

In another aspect, this document features methods of monitoring efficacyof a treatment for a condition of the cornea of a subject. The methodsinclude determining a first score for the cornea of the subject,administering one or more treatments to the eye of the subject, anddetermining a second score for the cornea of the subject. The first andsecond scores can be determined by the methods described herein. Themethods also include comparing the first and second scores. Changes inthe scores can be used as indications of whether or not the treatment iseffective. For example, a decrease in a score can indicate that thetreatment was effective, and no change or an increase in a score canindicate that the treatment was ineffective.

In another aspect, this document features methods of monitoringprogression of a condition of the cornea of a subject. The methodsinclude determining a first score for the cornea, and determining asecond score for the cornea. The first and second scores can bedetermined by the methods described herein. The methods also includecomparing the first and second scores. Changes in the scores can be usedas indications of whether or not the treatment is effective. Forexample, wherein a decrease from the first score to the second score canindicate that the condition is improving, no change between the firstand second scores can indicate that the condition is stable, and anincrease from the first score to the second score can indicate that thecondition is worsening.

Implementations of the above aspects can include one or more of thefollowing.

The selection can be received via an adjustable control overlaid on thedigital image of the cornea. Processing the selection can includedividing the selection into a plurality of zones, analyzing each of thezones to detect a presence of the one or more artifacts, and modifyingthe selection to exclude zones in which the presence of the one or moreartifacts are detected to define the evaluation area. The one or moreartifacts can include one of a specular reflection and a confocalreflection. The specular or confocal reflection can be detected using aflooding algorithm. Determining the amount of staining can includemapping an angle corresponding to the hue value to a scalar value withina predetermined range, and determining the amount of staining as aproduct of the scalar value and at least one component of the polarcoordinate based color space that is different from the hue. A paraboliccurve can be used in mapping the angle to the scalar value within thepredetermined range. The tracer material can be a fluorophore such asfluorescein. The digital image can be acquired in the presence of bluecobalt light. The score can indicate a degree of greenness of the imageof the cornea. The Cartesian color space can be the Red-Green-Blue (RGB)color space. The polar coordinate-based color space can be theHue-Saturation-Value (HSV) color space. The score and an association ofthe score with the digital image can be stored on a storage device. Thedigital images can represent corneas afflicted with a corneal epithelialdisease, dry-eye syndrome, ocular graft-versus-host disease, andSjogren's syndrome. The score being above a reference score can indicatethat the subject has a disease.

Particular implementations may realize none, one, or more of thefollowing advantages. Corneal images taken using different imagingsystems can be evaluated based on a standardized scale andquantitatively compared with one another. This can allow forrepeatability, consistency, and accuracy because the corneal images arenot scored based on subjective judgment of a human observer. The scoringsystem can be made sensitive to detecting small changes in cornealfluorescein staining that might not be detectable by a human observer.The methods and systems described herein thus provide an objectivetechnique that can be used, e.g., to diagnose and to monitor response totherapy, using a computer based analysis.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art. The materials, methods, and examples are illustrative only andnot intended to be limiting. All publications, patent applications,patents, sequences, database entries, and other references mentionedherein are incorporated by reference in their entirety. In case ofconflict, the present specification, including definitions, willcontrol.

Other features and advantages will be apparent from the followingdetailed description and figures, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an example of a system for calculatingscores for corneal images acquired through different imaging systems.

FIG. 2A depicts a Cartesian color space.

FIGS. 2B-2D depicts polar coordinate based color spaces.

FIG. 3A is a flowchart representing an example sequence of operationsfor determining a score for digital image of the cornea.

FIG. 3B is an example of a curve used in an operation depicted in FIG.3A.

FIG. 4 is a block diagram of an example of a computing system.

FIGS. 5A-5D is examples of corneal images for which scores have beencalculated.

DETAILED DESCRIPTION

Presented here are techniques that can be used to assess and quantifyconditions such as corneal epitheliopathy, which as used in thisdocument, broadly refers to afflictions or diseases of the epithelium.Corneal epitheliopathy manifests as focal punctate areas with a tendencyto converge. These areas can be identified from digital images acquiredafter staining the areas with a tracer material such as fluorescein. Theareas with epitheliopathy capture fluorescein dye instilled onto theocular surface and are manifested as focal punctuate areas. During aclinical exam, the clinician directs a blue cobalt light towards thecornea, thus stimulating the captured dye and producing a greenfluorescent signal. Grading the amount of fluorescence due tofluorescein is known as Corneal Fluorescein Staining (CFS) grading, andcan be used to evaluate diseases that affect the ocular surface.Examples of such diseases include corneal epitheliopathy, dry eye,ocular graft-versus-host-disease, Sjogren's syndrome, and corneal ulcerssecondary to various infections (e.g., herpes virus, bacterial).

Ophthalmologists often grade or quantify the amount of fluorescence incorneal images based on a visual inspection. In some cases, the imagesare visually compared to reference drawings or photographs to bin theminto various categories. For example, the National Eye Institute (NEI)provides a corneal fluorescein scoring system where the corneal surfaceis divided in five regions. Each region is graded with a score betweenone and three points based on the level of fluorescence in the region.The scores from the five regions are added, resulting in a maximum offifteen points for the most affected corneas. The threshold for athree-point score in a given region is relatively low and patients withmoderate conditions often receive a three point score for a givenregion. In addition, even if a highly affected area experiences moderateimprovement, the area may receive the same score that it did prior tothe improvement. Therefore, the system may not be sensitive enough todetect changes in moderate to severe conditions.

The above mentioned methods can therefore be subjective and prone toerrors. For example, the same image can be graded differently by twodifferent clinicians, or even when the same clinician grades the imageat two different times. In addition, human observers may also not beable to detect small changes, rendering such visual gradingcoarse-grained and prone to inaccuracies.

The technology described in this document allows computer-based analysisof digital images of the cornea such that images across differentimaging systems, conditions and time can be evaluated based on astandardized score assigned to each of the images. The techniquesdescribed in this document facilitate quantifying the fluorescence incorneal images, and objectively determining the condition of affectedareas on the corneal surface. By quantifying the intensity of thefluorescent signal, the techniques allow for distinguishing betweensevere epithelial defects that tend to capture more fluorescein andtherefore emanate stronger signals, from mild to moderate dry eyecorneal epitheliopathy that is diffuse and emanate weaker signals.Therefore, locations with severe damage can be identified even if thelocation is not spread over a large area of the cornea.

FIG. 1 shows a block diagram of an example of a system 100 forcalculating scores for corneal images acquired through different imagingsystems. The system 100 includes one or more imaging systems 105A, 105B,. . . , 105N (105, in general). The imaging systems 105A, 105B, . . . ,105N are used to acquire images sets 110 a, 110 b, . . . , 110 n,respectively (110, in general). The imaging systems 105 can be differentor substantially similar to one another. For example, the imaging system105A can be a slit-lamp camera and the imaging system 105B can be astandard digital photography camera. In another example, the imagingsystems 105A, 105B, . . . , 105N can all be slit-lamp cameras ofdifferent makes, models or have imaging parameters that are differentfrom one another. The corresponding image sets 110 acquired by thedifferent imaging systems 105 can therefore vary significantly from oneanother. For example, the images across the different imaging systems105 can vary from one another in resolution, white balance, lightingcharacteristics or other image parameters. In some implementations, forcorneal imaging, the imaging systems 105 can be configured to obtain theimages in the presence of light that excites the fluorophore used as thetracer material. For example, in case of Corneal Fluorescein Stainingimaging (in which fluorescein is used), the images can be acquired bythe imaging systems 105 in the presence of blue cobalt light. In suchcases, images taken by different imaging systems cannot be reliablycompared to one another based simply on visual inspection by a humanobserver. For example, if two corneal images taken by different imagingsystems are compared by a human observer, the perceived difference inthe imaged fluorescence may be due to inherent differences in theimaging hardware and/or software.

The corneal image sets 110 can be obtained, for example, byadministering a tracer material to the eye and imaging the eye underparticular lighting conditions. For example, the tracer material caninclude a fluorophore, i.e., a fluorescent chemical compound that canre-emit light upon light excitation. The fluorophore can include organiccompounds having aromatic groups, or plane/cyclic molecules with severala bonds.

The fluorophore used in staining the cornea often includes fluorescein.Fluorescein, a synthetic organic compound, is soluble in water andalcohol at least to some degree. Fluorescein is widely used as a tracermaterial because of having desirable fluorescence properties. In someimplementations, fluorescein sodium, a sodium salt of fluorescein, canbe used as a tracer material in the diagnosis of corneal abrasions,corneal ulcers and herpetic corneal infections. In some implementations,fluorescein or one of its derivatives can also be used in contact lensfitting to evaluate the tear layer under the lens.

In some implementations, the corneal images 110 acquired by the sameimaging system 105 can vary from one another. For example, if the imagesare taken some time apart, variability due to, for example, parameterdrift, or different lighting conditions can contribute to thevariability of the images. In some implementations, the techniquesdescribed herein can be used to analyze corneal images from differentclinical centers and patients, to compare them quantitatively. Thetechniques can also be used, for example, to optimize CFS evaluation inclinical trials. The manual grading systems typically used in controlledtrials may lack adequate resolution for detecting changes (improvementor worsening) in corneal staining after a treatment is administered. Asa result, advanced clinical trials for drugs often fail because of alack of clear determination of disease endpoint achievements. Thetechniques described herein can be used to evaluate corneal imagesobjectively, thereby allowing for accurate assessments of changesattributable to a drug or treatment regimen.

The system 100 includes a score calculator module 115 that can be usedto determine or assign a score to the corneal images 110 or to portionsthereof. The score calculator module 115 can be implemented on acomputing device and configured to account for variability that existsin images acquired using one or more imaging systems 105. In someimplementations, the score calculator module 115 can be implementedusing a general purpose computer such as a desktop or laptop computer ora mobile device that is capable of executing one or more softwareprograms. In some implementations, the score calculator module 115 isconfigured to execute one or more image processing application programssuch as ImageJ, developed at the National Institutes of Health. In someimplementations, the score calculator module may be implemented as aplug-in (e.g., a Java-based plug-in) for Image J or other image analysistool.

In some implementations, the score calculator module 115 includes a userinterface 118 that is configured to accept user input as well as providescore outputs to a user. In some implementations, a user can interactwith the score calculator module 115 through the user interface 118. Insome implementations, the score user interface 118 can include anadjustable control over overlaid on an image. For example, the userinterface 118 can include a circle (or another shape) that is overlaidon a corneal image. The user may be allowed to adjust the circle to fitthe corneal size. The user interface 118 therefore provides theflexibility of a user choosing the corneal shape to facilitate a moreaccurate score calculation.

In some implementations, the score calculator module 115 can beconfigured to process the corneal image to exclude areas that includeundesirable imaging artifacts, which may lead to inaccuracies in thecalculated score. Examples of such artifacts include specularreflections and confocal reflections. A specular reflection in a cornealimage occurs due to reflection of light from the surface of the eye, andis manifested as one or more spots of high intensity within the cornealimage, and do not represent the true colors of the corneal image.

The score calculator module 115 can employ various processes to detectthe areas that include the undesirable imaging artifacts. For example,the areas with specular reflections can be detected using a floodingalgorithm. For example, pixels that satisfy threshold conditions on thevalues of one or more of the R, GC and B components can be detected, andthe pixel value can be set at an initial high value. A contiguous areacan then be selected such that the pixel values within the contiguousarea are within a certain tolerance range above or below the initialvalue. The area can then be flagged (for example, by setting the colorof the pixels as red) as specular reflections. In some implementations,the process can be repeated over the entire image to detect specularreflections or other artifacts.

In some implementations, a grid can be overlaid on the corneal image todivide the corneal image into a plurality of areas. For example, the NEIcorneal fluorescein staining scoring system grid, which divides thecorneal area into five zones or areas can be used. Other types of gridswith more or less number of zones can also be used. The areas that aredetermined to include the undesirable artifacts can then be excludedfrom subsequent calculations to determine the score for the cornealimage. An evaluation area can be defined upon exclusion of the areaswith the undesirable artifacts. In some implementations, where thecorneal image does not include any undesirable artifact, the entireimage of the cornea may be included within the evaluation area.

In some implementations, the score calculator module 115 calculates ascore for the evaluation area in accordance with one or more imageanalysis techniques described below. In some implementations, the scorefor a given corneal image is determined as a sum of the scores for theindividual zones, except the zones that are excluded because of havingone or more artifacts. In some implementations, only the particularpixels that represent artifacts are excluded from the score calculationfor the entire corneal image. The image analysis techniques can includedetermining color information from pixels of the evaluation area. Ingeneral, the score calculator module 115 assigns scores to the cornealimages 110 or portions thereof and outputs an image set 120 in whicheach image is associated with a standardized score. For example, theimage set 120 can include one or more corneal images 110 that areassigned a corresponding score based on the amount of fluoresceinstaining detected in the image. The images from the set 120 and anassociation with the respective scores can be stored in a storagedevice.

The methods and systems described herein process digital images orportions thereof based on their color properties. Color properties canbe described, for example, using color spaces that represent colors astuples of numbers, typically as three or four values or colorcomponents. Examples of color spaces include RGB, CMY, CMYK, YIQ, YUV,YCrCb, HSV, HSI, IHC and HSL color spaces. In general, color spaces canbe broadly classified into Cartesian and polar coordinate based colorspaces. An understanding of such color spaces is important in themethods and systems described herein and are therefore described nextwith reference to FIGS. 2A-2D.

Referring now to FIG. 2A, an RGB color space is shown as an example of aCartesian color space. In this color space, a color is represented in athree dimensional Cartesian space composed on three colors red, greenand blue. The RGB color space is an additive color model in which red,green, and blue colors are added together in various ways to reproduce abroad array of colors. The RGB color space is typically used forsensing, representation, and display of images in electronic systems,such as televisions, digital cameras, computers and handheld mobiledevices. In the example shown in FIG. 2A, different colors are encodedusing three 8-bit unsigned integers (0 through 255) representing theintensities of red, green, and blue. This representation is the currentmainstream standard representation in image file formats such as JPEG orTIFF. Such encoding of the RGB space results in more than 16 milliondifferent possible colors. As shown in FIG. 2A, the colors at thevertices of the RGB color space may be represented as the followingpoints: (0, 0, 0) is black, (255, 255, 255) is white, (255, 0, 0) isred, (0, 255, 0) is green, (0, 0, 255) is blue, (255, 255, 0) is yellow,(0, 255, 255) is cyan and (255, 0, 255) is magenta. Any point in thevolume bounded by these vertices represents a mixed color that can bebroken down into red, green and blue components and represented in theRGB space as a point (r, g, b). Further, lines and planes may also bedefined in the RGB color space. For example, the line connecting pureblack (0, 0, 0) and pure white (255, 255, 255) may be defined as a grayline 205. Other examples of Cartesian color spaces include the YIQ, YUVand YCbCr spaces.

The Cartesian color spaces, while ideal for describing colors in digitalformats, are not well suited for describing colors that are practicalfor human interpretation. For example, human beings do not perceive acolor in terms of its component primary colors. Rather, humans usuallydescribe a color by its hue, saturation and brightness or intensity. Hueis an attribute that describes what a color actually is (for example,red, yellow, orange, and cyan), whereas saturation is a measure thatdescribes to what extent the color is diluted by white light. Brightnessis a descriptor that embodies the achromatic notion of intensity and isan important factor in describing color perception. Color spaces basedon these attributes of colors are ideal for algorithms related to humanperception of color, such as described herein. The IHC (Intensity, Hue,Chroma) color space described with respect to FIG. 2B is an example ofsuch a color space.

Referring to FIG. 2B, the IHC color space includes of a verticalintensity axis 215 and loci 220 a, 220 b (220 in general) of colorpoints that lie on planes perpendicular to the axis. The hue (H) 225 ofa color point within a locus plane (220 a for example) is represented byan angle with respect to a reference point while a chroma (C) 230 isrepresented as a linear distance of the point from the point ofintersection of the locus plane 220 a with the intensity axis 215. Eventhough, the example in FIG. 2B shows the loci 220 to be circular inshape, other polygonal shapes, such as triangles, pentagons, hexagonsetc., may be used to represent the loci. The area of the loci 220 is afunction of the intensity. In other words, the range of chroma is alsodependent on the intensity. For example, at zero intensity (i.e., I=0),all colors have zero chroma value and converge to black. Similarly, forthe maximum intensity (e.g., I=1), all colors have zero chroma value andconverge to white. Within this range, the area of the loci 220 (or therange of chroma values) may increase, for example from I=0 to I=0.5 andthen decrease again from I=0.5 to I=1. FIG. 2B shows the locus 220 bcorresponding to intensity I=0.75. For a given locus plane 220, the hueof a color point is determined by an angle from a reference point. Inthis example, red designates the reference point, i.e. zero hue, and thehue increases in a counterclockwise direction from the reference point.Other polar coordinate based color spaces, such as the HSL (Hue,Saturation, Lightness) and HSV (Hue, Saturation, Value) color spaces,also follow similar principles with hue being represented as an angle inan polar coordinate based coordinate system.

Referring now to FIG. 2C, a HSL color space also includes of a verticalaxis and loci 220 of color points that lie on planes perpendicular tothe axis. In this color space, the vertical axis represents lightness(L) 234. The HSL color space is also referred to HLS or HSI with Istanding for intensity. The HSL color space represents colors as pointsin a cylinder 231 (called a color solid) whose central axis 234 rangesfrom black at the bottom to white at the top, with colors distributedbetween these two extremities. The angle around the axis corresponds tothe hue 225, the distance of a point on a given locus 220 from the axiscorresponds to the saturation 232, and the distance along the axis 234corresponds to lightness or intensity. Unlike the chroma 230 in the IHCcolor space (FIG. 2A), the range of the saturation 232 is not a functionof the lightness or intensity.

Referring now to FIG. 2D, an example of a HSV color space representscolors via an inverted color cone 238 on a cylinder 240. Otherrepresentations of the HSV color space are also possible. In thisexample, the HSV color space includes a common vertical axis 236 for thecone 238 and the cylinder 240. The central axis 236 ranges from black atthe bottom to white at the top, with colors represented in loci 220distributed between these two extremities. The angle around the axiscorresponds to the hue 225, the distance of a point on a given locus 220from the axis corresponds to the saturation 232, and the distance alongthe axis 234 corresponds to the value V. The value can be scaled to bebetween 0 and 1. In this color space, the saturation 232 is a functionof the value V when V is between 0.5 and 1. For example, when V=1, allcolors converge to pure white. When V is between, 0 and 0.5, the rangeof the saturation is constant and not a function of the value, as shownin FIG. 2D.

In some implementations, hue information from digital images are used inthe methods and systems described herein. In some implementations, colorinformation corresponding to pixels in a digital image are converted toa polar coordinate based color space in determining a score thatrepresents a particular color content. For example, in determining aredness value for a portion of a digital eye image, the colorinformation from the pixels can be converted from the RGB color space tothe HSV color space and the hue information can be used in calculatingthe redness score of the portion. As described with respect to FIG. 2B,in general, hue is an attribute of polar coordinate based color spaceswhile most digital images are represented using Cartesian coordinatesystems such as the RGB color model. The RGB color information may betransformed into a polar coordinate based color space such as the HSIcolor space. For example, the hue may be calculated as:

$\begin{matrix}{H = \left\{ \begin{matrix}\theta & {B \leq G} \\{{360 - \theta},} & {B > G}\end{matrix} \right.} \\{{{where}\mspace{14mu} \theta} = {\cos^{- 1}\left\{ \frac{\frac{1}{2}\left\lbrack {\left( {R - G} \right) + \left( {R - B} \right)} \right\rbrack}{\left\lbrack {\left( {R - G} \right)^{2} + {\left( {R - B} \right)\left( {G - B} \right)}} \right\rbrack^{1/2}} \right\}}}\end{matrix}$

The saturation component is given by:

$S = {1 - {\frac{3}{\left( {R + G + B} \right)}\left\lbrack {\min \left( {R,G,B} \right)} \right\rbrack}}$

The intensity of the component is given by:

$I = {\frac{1}{3}\left( {R + G + B} \right)}$

In some implementations, the RGB color information can be transformedinto the HSV color space using the following equations. For example, thevalue component V can be calculated as:

V=max(R,G,B)

The saturation component Scan be calculated as:

$S = {\frac{delta}{\max \left( {R,G,B} \right)}\left\{ \begin{matrix}{if} & {{\max \left( {R,G,B} \right)} \neq 0} \\{else} & {S = 0}\end{matrix} \right.}$

wherein

delta=max(R,G,B)−min(R,G,B)

The hue component H is given by:

$\quad\left\{ \begin{matrix}{{delta} \neq 0} & \left\{ \begin{matrix}{H = {\frac{{60 \times \left( \frac{G - B}{delta} \right)} + 360}{360}\left\{ {{{if}\mspace{14mu} {\max \left( {R,G,B} \right)}} = R} \right\}}} \\{H = {\frac{{60 \times \left( \frac{B - R}{delta} \right)} + 360}{360}\left\{ {{if}\mspace{14mu} {\max\left( {R,G,{B = G}} \right\}}} \right.}} \\{H = {\frac{{60 \times \left( \frac{R - G}{delta} \right)} + 360}{360}\left\{ {otherwise} \right\}}}\end{matrix} \right. \\{{delta} = 0} & \left\{ {H = 0} \right\}\end{matrix} \right.$

Referring now to FIG. 3, a flowchart 300 represents an example sequenceof operations for determining a score of a digital image. In someimplementations one or more of the operations can be executed at thescore calculator module 115 described with reference to FIG. 1.

The operations include obtaining a digital image of the cornea stainedwith a tracer material (302). The tracer material can include afluorophore such as fluorescein. The digital image can be obtained froman imaging system substantially similar to the imaging system 105described with reference to FIG. 1. In some implementations, the digitalimage can be acquired in the presence of light that excites thefluorophore used as the tracer material. For example, in case of CFSimaging (in which fluorescein is used), the digital image can beacquired in the presence of blue cobalt light. In some implementations,the digital image can be obtained substantially directly from theimaging system. In some implementations, obtaining the digital image caninclude retrieving the digital image from a storage device.

Operations can also include receiving a selection of a portion of theimage (304). The selection can be received, for example, through a userinterface substantially similar to the user interface 118 described withreference to FIG. 1. The selection operation can allow a user, forexample via an adjustable control overlaid on the digital image of thecornea, to accurately select a region of interest from the cornea. Thiscan be done, for example, by providing a circle (or another shape) thatis overlaid on the corneal image. The user may be allowed to adjust thecircle (or any other shape that is provided) to accurately select aregion of interest within the cornea. The region of interest can alsoinclude the entire cornea. The interactive selection process also allowsfor repeatability to evaluate a substantially same area in differentimages.

Operations also include processing the selection to automaticallyexclude areas with one or more artifacts to define an evaluation area(305). The artifacts can include, for example, specular reflection andconfocal reflection. The artifacts such as specular or confocalreflections can be detected, for example, using a flooding algorithm,and the areas in which they are detected can be excluded from theevaluation area to improve the accuracy of the score. In someimplementations, the artifacts can also be determined using a thresholdfor pixel intensities. The threshold can be set, for example, by aclinician to avoid artifacts in the corneal images. In someimplementations, the threshold is set such that pixel intensities thatsatisfy a threshold condition are flagged as artifacts and thecorresponding zones or regions are excluded from the evaluation area.

For each of a plurality of pixels within the evaluation area, theCartesian color components are determined (306). For example, if thedigital image is represented using the RGB color space, the red, greenand blue components corresponding to the pixel value are determined. Insome implementations, the color components can be associated withanother Cartesian color space such as the CMY color space. In someimplementations, the plurality of pixels includes all pixels in theevaluation area. In some implementations, only a subset of the pixelswithin the evaluation is considered in calculating the score.

Operations further include determining a hue value from the Cartesiancolor components (308). As described above with reference to FIGS.2B-2D, hue is a component of a polar coordinate based color space andtherefore determining the hue value can include converting the Cartesiancolor components to a polar coordinate based color space. The polarcoordinate based color space can include, for example, the HSV colorspace. Converting the Cartesian color components to a polar co-ordinatebased color space can be done, for example, using the conversionsdescribed above with reference to FIGS. 2A-2D. The hue value can bedescribed in terms of an angle also as described with reference to FIGS.2B-2D.

Operations also include determining an amount of staining for each ofthe plurality of pixels (310). In some implementations, determining theamount of staining includes mapping an angle corresponding to the huevalue (e.g., between 0 and 360°) to a scalar value within apredetermined range (e.g., 0 and 1). For example, in determining amountof fluorescein staining, the hue values corresponding to a greenness ofthe image may lie within 20° and 220°. In some implementations, theangle values can be mapped to a scalar range between, for example, 0and 1. The mapping may be linear or non-linear. FIG. 3B shows an exampleof using a non-linear curve such as a parabola 320 to map hue valuesbetween 20° and 220° (represented along the x-axis) to a scalar rangebetween 0 and 1 (represented along the y-axis). In some cases, a linearfunction may also be used. For example, in determining an amount ofstaining in CFS images, a linear function may work better between 200°and 220° to avoid any confusion between the aqua or cobalt blue (colorof the light source utilized to excite fluorescein) and the green (colorof actual fluorescence representing epithelial defects). Other angleranges can be used for other fluorophores. In some implementations, theangle range that is mapped on to the scalar value can be selected basedon a location of a color in the hue circle. For example, in measuring anamount of staining in images related to CFS, an angle rangecorresponding to observed shades of green (or green-yellow) can beselected.

In some implementations, the scalar value itself can be used torepresent the amount of staining. In some implementations, the valuerepresenting the amount of staining (also referred to as stainingintensity) for a given pixel is determined as a product of the scalarvalue and one or more components of the polar coordinate based colorspace. In some implementations, when the HSV color space is used formeasuring fluorescein staining, the staining intensity can be referredto as greenness, and determined as a product of the scalar value withone or both of the S and V components. For example, the greenness of apixel can be computed as:

greenness=H×S×V.

If each of the H, S, and V components are normalized to be in the range[0,1], the greenness score is also within the range of [0, 1]. In suchcases, a score of zero implies that there is no greenness in theparticular pixel, whereas a score of one implies that the pixel iscompletely green. In some implementations, a threshold may be applied todetermine whether a particular pixel represents staining. For example,if a threshold of 0.018 is applied, a given pixel is determined torepresent staining only if the greenness value is higher than 0.018. Insome implementations, the threshold may be defined by a clinician (e.g.,an ophthalmologist) or empirically determined using a set of controlimages.

In some implementations, an upper limit may also be defined for thestaining intensity (e.g., greenness). In such cases, the stainingintensity corresponding to a perceived high amount of staining can beused as the upper limit. For example, if the greenness corresponding tosuch perceived high amount of staining is 0.4, any greenness score equalto or higher than 0.4 can be assigned the maximum possible value. Anyvalues equal to or greater than 0.4 can then be assigned a value of 1,i.e., the maximum possible value. In some implementations, greennessvalues in the range [0, 1] can be remapped to a wider range such as [0,100].

Operations further include assigning a score to the evaluation area(312). In some implementations, the score is determined as an average ofthe staining intensities corresponding to the plurality of pixels in theevaluation area. In some implementations, other measures of centraltendency such as weighted average, median value or mode can also be usedin determining the score. In some implementations, the score can bescaled to a value within a predetermined range (e.g. [0, 100]) beforebeing assigned to an evaluation area. The predetermined range can bechosen based on, for example, the type of image or application. In someimplementations, the scaling can be such that the highest determinedscore maps on to the upper end of the predetermined range (100, in thisexample) and the lowest determined score maps on to the lower end of thepredetermined range (0, in this example). The score is then mapped on toan appropriate value within the predetermined range. In someimplementations, the predetermined range can be fixed based onpredetermined high and low color values. In such cases, color valueshigher than the highest predetermined value are mapped on to the upperend of the range and color values lower than the lowest predeterminedvalue are mapped on to the lower end of the range. In someimplementations, the determined score is saved in a storage device alongwith an association that links the score with the corresponding image.

In some implementations, assigning the score to the evaluation area alsoincludes generating a table of percentage of a staining area. In someimplementations, a threshold for the staining intensity can be used todetermine whether or not a particular pixel is part of a staining area.For example, a given zone or area of the cornea is determined to be 20%stained if 20% pixels in the zone or area have staining intensities thatexceed the threshold. The threshold can be clinician-defined orempirically determined.

The staining in corneal images can therefore be represented both interms of stained area and staining intensity. For example, if a givenzone has a low average staining intensity, but a high percentage ofstaining area, the zone may be determined to have widespread superficialepithelial damage. In another example if a given zone has a lowpercentage of staining, yet pockets of high staining intensities, thezone may be determined to have localized but deep lesions.

In some implementations, the obtained digital image may be subjected toone or more pre-processing operations prior to calculating the score.The pre-processing operations can include, for example, noise removal orwhite balancing.

FIG. 4 is a schematic diagram of a computer system 400. The system 400can be used for the operations described with reference to theflowcharts in FIG. 3A. The system 400 can be incorporated in variouscomputing devices such as a desktop computer 401, server 402, and/or amobile device 403 such as a laptop computer, mobile phone, tabletcomputer or electronic reader device. The system 400 includes aprocessor 410, a memory 420, a computer-readable storage device 430(which may also be referred to as a non-transitory computer readablestorage medium), and an input/output device 440. Each of the components410, 420, 430, and 440 are interconnected using a system bus 450. Theprocessor 410 is capable of processing instructions for execution withinthe system 400. In one implementation, the processor 410 is asingle-threaded processor. In another implementation, the processor 410is a multi-threaded processor. The processor 410 is capable ofprocessing instructions stored in the memory 420 or on the storagedevice 430 to display graphical information for a user interface on theinput/output device 440.

The memory 420 stores information within the system 400. In someimplementations, the memory 420 is a computer-readable storage medium.The memory 420 can include volatile memory and/or non-volatile memory.The storage device 430 is capable of providing mass storage for thesystem 400. In one implementation, the storage device 430 is acomputer-readable medium. In various different implementations, thestorage device 430 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 440 provides input/output operations for thesystem 400. In some implementations, the input/output device 440includes a keyboard and/or pointing device. In some implementations, theinput/output device 440 includes a display unit for displaying graphicaluser interfaces. In some implementations the input/output device can beconfigured to accept verbal (e.g. spoken) inputs.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, or in combinations ofthese. The features can be implemented in a computer program producttangibly embodied in an information carrier, e.g., in a machine-readablestorage device, for execution by a programmable processor; and featurescan be performed by a programmable processor executing a program ofinstructions to perform functions of the described implementations byoperating on input data and generating output. The described featurescan be implemented in one or more computer programs that are executableon a programmable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. A computer program includes a set ofinstructions that can be used, directly or indirectly, in a computer toperform a certain activity or bring about a certain result. A computerprogram can be written in any form of programming language, includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor will receive instructionsand data from a read-only memory or a random access memory or both.Computers include a processor for executing instructions and one or morememories for storing instructions and data. Generally, a computer willalso include, or be operatively coupled to communicate with, one or moremass storage devices for storing data files; such devices includemagnetic disks, such as internal hard disks and removable disks;magneto-optical disks; and optical disks. Storage devices suitable fortangibly embodying computer program instructions and data include allforms of non-volatile memory, including by way of example semiconductormemory devices, such as EPROM, EEPROM, and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in, ASICs(application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube),LCD (liquid crystal display) monitor, eInk display or another type ofdisplay for displaying information to the user and a keyboard and apointing device such as a mouse or a trackball by which the user canprovide input to the computer.

The features can be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include, e.g., a LAN, a WAN, and thecomputers and networks forming the Internet.

The computer system can include clients and servers. A client and serverare generally remote from each other and typically interact through anetwork, such as the described one. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

The processor 410 carries out instructions related to a computerprogram. The processor 410 may include hardware such as logic gates,adders, multipliers and counters. The processor 410 may further includea separate arithmetic logic unit (ALU) that performs arithmetic andlogical operations.

The methods and systems described herein can be used in a number ofclinical applications. For example, a score can be used to determine theseverity of a disorder associated with the presence of the stainingintensity. Taking corneal epitheliopathy as an example, the presence ofwhich is associated with a number of conditions (including, but notlimited to dry eye syndrome, graft-versus-host disease, Sjogren'ssyndrome, allergies, corneal infections, etc.), a higher scoredetermined by a method described herein can be associated with greaterseverity of the condition. A lower score can indicate that the conditionis less severe.

The methods can also be used to monitor progression or treatment of acondition. For example, a first score and/or percentage area aredetermined at a first time point, e.g., before or during administrationof a treatment for the condition associated with the presence of thestaining intensity, and a second score is determined at a later time.Again taking corneal epitheliopathy as an example, a first score and/orpercentage area are determined at a first time point, e.g., before orduring treatment, and a second score and/or percentage area aredetermined at a later time point. The two scores can then be compared,and an increase in the score can indicate progression (i.e., worsening)of the condition or a lack of efficacy of the treatment; no change canindicate that any treatment has at best stopped progression (in aprogressive disorder), or has been ineffective; and a decrease in thescore can indicate that the treatment has been effective. An increase inthe percentage area can indicate that the condition is spreading,whereas a decrease can indicate that the spread area is reducing.

Examples

The methods and systems described herein are further described using thefollowing examples (with reference to FIG. 5A-5D), which do not limitthe scope of the claims. FIGS. 5A-5D illustrate score and areadetermination for fluorescein staining. The images were obtained frompatients with dry eye disease. A Haag-Streit BQ 900 IM imaging systemwas used to acquire the images. Fifty images from patients with dry eyedisease were analyzed, and a strong correlation was found between thescores of NEI corneal fluorescein staining and the scores computed usingthe technique described above. A metric known as the Spearman'scoefficient of correlation (R) was computed based on the data. Thecoefficient of correlation is a metric that represents the degree ofco-variation between two variables, and indicates the degree to whichtwo variable's movements are associated. The value of R varies between−1.0 and 1.0. An R value of 1.0 indicates that there is perfectcorrelation between the scores of both variables. An R value of −1.0indicates perfect correlation in the magnitude of the scores of bothvariables but in the opposite direction. On the other hand, an R valueof 0 indicates that there is no correlation between the scores of thetwo variables. The R value between the scores of NEI corneal fluoresceinstaining and the scores computed using the technique described above wasequal to 0.8 with statistical significance represented as p<0.001. Astrong correlation was also found between the NEI score and theintensity obtained with the techniques described above (R=0.85;p<0.001). Therefore, while following the trend of the NEI scoresmanually provided by a physician, the grading techniques describedherein provides an accurate and high resolution scale that does not relyon human interpretation or arbitrary scoring criteria.

FIG. 5A shows the image of an eye and FIG. 5B shows the correspondingCFS image. The specular reflection 503 in the image of FIG. 5A isrepresented as the red spot 504 in the CFS image of FIG. 5B. The areaincluding the specular reflection 503 is therefore excluded from theevaluation area. The inferior area 502 of the images in FIGS. 5A and 5Bwas used as the evaluation area and the image in FIG. 5B scored 84.2with an average level of fluorescence intensity of 22.

FIG. 5C shows the image of another eye and FIG. 5D shows thecorresponding CFS image. The specular reflections 506 and 507 in theimage of FIG. 5A are manifested as the spots 508 and 509, respectivelyin the CFS image of FIG. 5D. The area including the specular reflections506 and 507 are therefore excluded from the evaluation area. Theinferior area 505 of the images in FIGS. 5A and 5B was used as theevaluation area and the image in FIG. 5D scored 100 with an averagelevel of fluorescence intensity of 90.

The algorithm was implemented on the Java-based imaging-processingplatform ImageJ (National Institutes of Health; Rasband, ImageJ, U. S.National Institutes of Health, Bethesda, Md., USA (imagej.nih.gov/ij/),1997-2011; Abramoff et al., Biophotonics International (11)7:36-42(2004)) as a plug-in.

Other Embodiments

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. For example,elements of one or more implementations may be combined, deleted,modified, or supplemented to form further implementations. As yetanother example, the logic flows depicted in the figures do not requirethe particular order shown, or sequential order, to achieve desirableresults. In addition, other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

1.-35. (canceled)
 36. A computer-implemented method for determining ascore representing an amount of staining of a cornea, the methodcomprising: obtaining a digital image of the cornea stained with atracer material; processing, by one or more processing devices, thedigital image to exclude areas with one or more artifacts, whereinprocessing the digital image comprises: dividing the digital image intoa plurality of zones, and excluding zones in which a presence of the oneor more artifacts are detected to define a plurality of pixels; for eachpixel of the plurality of pixels: determining a plurality of colorcomponents that are based on a Cartesian color space, determining, fromthe color components, a hue value in a polar coordinate based colorspace, and determining an amount of staining of the cornea as a functionof the hue value; and assigning a score to the digital image based onthe amount of staining calculated for the plurality of pixels, whereinthe score indicates a degree of greenness of the image of the cornea.37. The method of claim 36, wherein the one or more artifacts includeone of a specular reflection and a confocal reflection.
 38. The methodof claim 36, wherein determining the amount of staining furthercomprises: mapping an angle corresponding to the hue value to a scalarvalue within a predetermined range; and determining the amount ofstaining as a product of the scalar value and at least one component ofthe polar coordinate based color space that is different from the hue.39. The method of claim 36, wherein the tracer material is afluorophore.
 40. The method of claim 39, wherein the fluorophore isfluorescein.
 41. The method of claim 40, wherein the digital image isacquired in a presence of blue cobalt light.
 42. The method of claim 36,wherein the Cartesian color space is an RGB color space.
 43. The methodof claim 36, wherein the polar coordinate based color space is an HSVcolor space.
 44. The method of claim 36, wherein the digital imagesrepresent corneas afflicted with a corneal epithelial disease.
 45. Acomputer readable storage device having encoded thereon computerreadable instructions, which when executed by a processor, cause aprocessor to perform operations comprising: obtaining a digital image ofa cornea stained with a tracer material; processing the digital image toexclude areas with one or more artifacts, wherein processing the digitalimage comprises: dividing the digital image into a plurality of zones,and excluding zones in which a presence of the one or more artifacts aredetected to define a plurality of pixels; for each of a plurality ofpixels: determining a plurality of color components that are based on aCartesian color space, determining, from the color components, a huevalue in a polar coordinate based color space, and determining an amountof staining of the cornea as a function of the hue value; and assigninga score to the digital image based on the amount of staining calculatedfor the plurality of pixels, wherein the score indicates a degree ofgreenness of the image of the cornea.
 46. The computer readable storagedevice of claim 45, wherein determining the amount of staining furthercomprises: mapping an angle corresponding to the hue value to a scalarvalue within a predetermined range; and determining the amount ofstaining as a product of the scalar value and at least one component ofthe polar coordinate based color space that is different from the hue.47. The computer readable storage device of claim 45, wherein thedigital images represent corneas afflicted with a corneal epithelialdisease.
 48. The computer readable storage device of claim 45, whereinthe tracer material is a fluorophore.
 49. The computer readable storagedevice of claim 48, wherein the fluorophore is fluorescein.
 50. Thecomputer readable storage device of claim 49, wherein the digital imageis acquired in a presence of blue cobalt light.
 51. The computerreadable storage device of claim 45, wherein the Cartesian color spaceis an RGB color space, and the polar coordinate based color space is anHSV color space.
 52. A system for determining a score representing anamount of staining of a cornea, the system comprising: an imaging systemconfigured to obtain a digital image of the cornea stained with a tracermaterial; and a score calculator module configured to: process thedigital image to exclude areas with one or more artifacts, theprocessing comprising: dividing the digital image into a plurality ofzones, and excluding zones in which a presence of the one or moreartifacts are detected to define a plurality of pixels, for each of aplurality of pixels: determining a plurality of color components thatare based on a Cartesian color space, determine, from the colorcomponents, a hue value in a polar coordinate based color space, anddetermine an amount of staining of the cornea as a function of the huevalue, and assign a score to the digital image based on the amount ofstaining calculated for the plurality of pixels, wherein the scoreindicates a degree of greenness of the image of the cornea.
 53. Thesystem of claim 52, further comprising a storage device configured tostore the score and an association of the score with the digital image.54. The system of claim 52, wherein the tracer material is a fluorophorethat includes fluorescein.
 55. The system of claim 52, wherein theimaging system further comprises a light source for radiating bluecobalt light to illuminate the cornea.